Image fusion with automated compensation for brain deformation

ABSTRACT

A system can include a model to represent a volumetric deformation of a brain corresponding to brain tissue that has been displaced by at least one of disease, surgery or anatomical changes. A fusion engine can perform a coarse and/or fine fusion to align a first image of the brain with respect to a second image of the brain after a region of the brain has been displaced and to employ the deformation model to adjust one or more points on a displacement vector extending through a displaced region of the brain to compensate for spatial deformations that occur between the first and second image of the brain.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional PatentApplication 61/735,101 filed on Dec. 10, 2012, and entitled IMAGE FUSIONWITH AUTOMATED COMPENSATION FOR BRAIN DEFORMATION, the entirety of whichis incorporated by reference herein.

TECHNICAL FIELD

This disclosure relates to image fusion with automated compensation forbrain deformation.

BACKGROUND

Image fusion has become a common term used within medical diagnosticsand treatment. The term is used when multiple patient images areregistered and overlaid or merged to provide additional information.Fused images may be created from multiple images from the same imagingmodality, or by combining information from multiple modalities, such asmagnetic resonance image (MRI), computed tomography (CT), positronemission tomography (PET), and single photon emission computedtomography (SPECT), for example. In radiology and radiation oncology,these images serve different purposes. For example, CT images are usedmore often to ascertain differences in tissue density while MRI imagesare typically used to diagnose brain tumors.

For accurate diagnoses, radiologists must integrate information frommultiple image formats. Fused, anatomically-consistent images areespecially beneficial in diagnosing and treating cancer. Many companieshave recently created image fusion software for both improved diagnosticreading and for use in conjunction with radiation treatment planningsystems. With the advent of these new technologies, radiationoncologists can take advantage of intensity modulated radiation therapy(IMRT). Being able to overlay diagnostic images onto radiation planningimages results in more accurate IMRT target tumor volumes, for example.

Another area where one type of image is fused on to another includes theuses of atlases in fusion. For instance, medical images can varysignificantly across individuals due to people having organs ofdifferent shapes and sizes. Therefore, representing medical images toaccount for this variability is crucial. A popular approach to representmedical images is through the use of one or more atlases. Here, an atlasrefers to a specific model for a population of images with parametersthat are learned from a training dataset. One example of an atlas is amean intensity image, commonly referred to as a template. However, anatlas can also include richer information, such as local imagestatistics and the probability that a particular spatial location has acertain label. New medical images can be mapped to an atlas, which hasbeen tailored to the specific application, such as segmentation andgroup analysis. Mapping an image to an atlas usually involvesregistering the image and the atlas. This deformation can be used toaddress variability in medical images.

SUMMARY

This disclosure relates to image fusion with automated compensation fortissue deformation.

In one example, a system can include a model to represent a volumetricdeformation of a brain corresponding to brain tissue that has beendisplaced or shifted by at least one of disease, surgery or anatomicalchanges. A fusion engine can perform a coarse fusion to align a firstimage of the brain with respect to a second image of the brain after aregion of the brain has been displaced and to employ the model to adjustone or more points on a displacement vector extending through adisplaced region of the brain to compensate for spatial deformationsthat occur between the first and second image of the brain.

As another example, a method can include determining an initial volumefor at least one selected anatomic region in a first three-dimensionalbrain image for a patient. Another volume for the at least one selectedanatomic region can be determined in a second three-dimensional brainimage for the patient. Shrinkage or growth factors can be computed fromthe first three-dimensional brain image based on a difference betweenthe initial volume and the another volume for the at least one selectedanatomic region. A mapping between the first and secondthree-dimensional brain images can be generated based on the computedshrinkage or growth factors and at least one volume parameter for thebrain that compensates for distortion between the first and secondthree-dimensional brain images.

As yet another example, a method can include accessing a first imagedata of an anatomical feature and second image data of the anatomicalfeature. A non-rigid alignment can be performed between the first imagedata and the second image data. A vector field mapping can be determinedbased on the non-rigid alignment to provide displacement data to a userregarding displacement between the first image and the second imagedata.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a system to perform image fusion whilecompensating for anatomical deformations in images.

FIG. 2 illustrates an example of a brain deformation in simplified brainimages demonstrating a determination of displacement vectors andfiducials.

FIG. 3 illustrates an example method for performing image fusion betweenanatomical images taken before, during, and after operative procedures.

FIG. 4 illustrates an example method for performing image fusion betweenan atlas image and brain images taken before, during, and afteroperative procedures.

FIG. 5 illustrates an example of fusion engine to perform rigid andnon-rigid alignment between preoperative and postoperative images togenerate displacement feedback with respect to a selected point withinthe images.

FIG. 6 illustrates example images of a preoperative image, a rigidlyaligned image, and a non-rigidly aligned image.

FIG. 7 illustrates an example method for performing displacement mappingvia rigid and non-rigid alignment or preoperative and postoperativeimages.

FIG. 8 illustrates an example system for performing fusion withcompensation for brain deformation.

DETAILED DESCRIPTION

This disclosure relates to systems and methods to improve image fusiontechnologies where non-homogenous compensation methods are applied toimages having post-procedural brain deformations to improve overallfusion performance. The approach disclosed herein can employ a workflowthat can initially involve a coarse fusion that can be performedutilizing various brain and spatial morphological and volumetricparameters before, during and/or after medical, diagnostic or surgicalprocedures have been performed. In some examples, coarse fusion of theoverall brain can be initially accomplished by fusing the skull,ventricular volume as well as the subarachnoid volume between images ofthe patient's brain and other brain images or a common image or atlasframework. Fiducial markers, such as identifiable landmarks, in thebrain can then be applied to the coarse fusion to further compensate andadjust for other distortions that may have occurred. Such fusion can beapplied to images from the same or different imaging technologies, suchas Computed Tomography (CT) and Magnetic Resonance Imaging (MRI).

As a further example, the methods can be applied to preoperative andpost-operative imaging to enable more accurate comparisons between suchimages and account for anatomical deformations that may have occurred,such as due to the respective procedures or other patient conditions(e.g., trauma, lesions or the like). In another example, thecompensation methods can be utilized to improve the fusion of atlaseswith brain images for patients that have undergone surgical proceduresor otherwise experience deformations in cranial and intracranialstructures. For instance, the fusion process enables more accurateregistration of the atlas and the brain image by adjusting the atlasbased on the computed resultant deformation. The fusion can be furtherimproved by utilizing additional landmarks in a target region ofinterest, thereby producing a more accurate brain atlas for the patient.As a result, the resulting brain atlas can further lead to improvedaccuracy in surgical procedures (e.g., stereotactic procedures) in whichthe target ROI resides in a spatially deformed region.

FIG. 1 illustrates an example of a system 100 that performs image fusionwhile compensating for deformations in images that may have occurred dueto operative procedures or other localized anatomical changes. In theexample of FIG. 1 and other examples disclosed herein, image fusion isoften discussed in the context of images for a patient's brain pre- andpost-procedure. However, it is to be understood that image fusion isapplicable to other images of a given anatomical structure that canexperience one or more localized deformations for any reasons, whetheror not a procedure has been performed. Thus, as used herein, apost-procedure image is intended to be interchangeable withpost-deformation image, meaning that at least one of the images beingfused has an anatomical deformation relative to at least another of theimages being fused regardless of what might have caused suchdeformation. Post-procedure is typically used herein in the descriptionand claims simply for sake of consistency and not by way of limitationunless explicitly specified.

The system 100 includes a fusion engine 110 that processes variousimages 120 and applies non-homogenous compensation methods to the imageshaving post-procedural brain deformations to improve overall fusionperformance. Fusion involves overlaying and co-registration (e.g.,alignment) of various images 120 by the fusion engine 110 (e.g.,software and/or processor performing fusion) and/or other data wheresome of the images may involve preoperative and postoperative views ofthe brain. Such fused images can be generated as a magnetic resonanceimage (MRI), computed tomography (CT), positron emission tomography(PET), and single photon emission computed tomography (SPECT), to namebut a few examples. In some examples, fused images can be generated at134 by the fusion engine 110 between the same imaging modalities (e.g.,preoperative/postoperative MRI on MRI) or can occur via differentimaging modalities (e.g., preoperative/postoperative MRI on CT). Inother examples, one or more of the images 120 could be an MRI or CTimage and another type of image could be an atlas of the brain (e.g.,coordinate point mapping of the brain), wherein the atlas image 120 isto be registered with a postoperative brain image 120.

During (e.g., perioperatively) and after a surgical procedure on thebrain, image compensation can be applied to perioperatively acquiredimages 120 and can initially involve a coarse fusion (e.g., rigidalignment between images) to account for brain deformations and adjustbrain images accordingly. As used herein, the term “coarse” relates toan initial adjustment of a deformed image wherein volume parameterscomputed for the brain are used to adjust the respective image (e.g.,increase or decrease a deformed boundary region in an image based onvolumetric computations). The coarse fusion can be performed by adeformation model 130 (also referred to as a deformation model). Thedeformation model 130 utilizes various brain volumetric parameters,shown as Vol 1 through Vol N, N being a positive integer, which accountsfor volume differences in brain physiology before and after surgicalprocedures have been performed. For example, the deformation model 130can be generated by analyzing a plurality preoperative and postoperativeimage sets of brain images that have been captured before procedureshave taken place and after the respective procedures. The deformationmodel 130 can model volumetric differences that are observed over theplurality of image sets. This can include the computation of vectorfields having a magnitude and direction with respect to the preoperativeand postoperative images. The deformation model 130 can also considerthe type of procedure performed, the angle and depth of insertion of agiven surgical probe, and how surgical deformations can change overtime. A comparison of the pre- and post-surgical volumes can be employedto compute growth or shrinkage coefficients that can be stored as thevolumetric parameters. In one example, the volumetric parameters caninclude a ventricular region and, in another example, such parameterscan include a subarachnoid region. The coefficients and other volumetricparameters can be utilized to fit volumes of interest (e.g., ventriclesand cortex) of the atlas to the patient's anatomy.

The fusion engine 110 can also employ fiducial markers (e.g., imagenotations or points of interest) and/or known landmarks 140 on the brainimage 120 that has undergone the coarse fusion to further compensate andadjust for other distortions that may have occurred during and/or afterthe procedure, corresponding to a fine fusion (e.g., non-rigidalignment). The fiducial markers and landmarks 140 can be appliedmanually by an operator and/or can be applied automatically (e.g., by analgorithm or learning system). Such compensation can be applied tovarious imaging technologies as noted above, wherein the compensationmethods can be applied to perioperative imaging to enable more accuratecomparisons between such images and account for deformations that mayhave occurred due to the respective procedures. In another example, thecompensation methods can be utilized to improve the registration ofatlases on to brain images 120 that have undergone surgical proceduresor otherwise experienced one or more anatomical deformations, whereinthe compensation by the fusion engine 110, deformation model 130, andmarkers 140 enables more accurate overlay of the atlas on to the brainimage by adjusting the atlas in accordance with the resultantdeformation.

In some examples, the system 100 can be employed to fuse brain imagingof patients before and after a mass occupying lesion has occurred. Asused herein, a mass occupying lesion can refer to a presence or absencestructure or fluid that cause a distortion or deformation of one or morespatial region of the brain. The distortion or deformation can be staticor it may evolve over time and be tracked by a sequence of imagesacquired at different times. The system 100 may be used for severalsurgical and non-surgical uses but for purposes of brevity, examples ofstereotactic procedures such as deep brain stimulation are used toexplain applicable methods. The fusion engine 110 can be employed toco-register images 120 acquired intraoperatively or postoperatively,including before and after pneumocephalus, for example, has occurred.Additional imaging can be performed to track deformation resulting fromthe pneumocephalus. The resulting atlas that is fused to theperioperative image(s) can be utilized to aid in accurate placement ofdeep brain stimulation leads or related procedures.

The system 100 can generate image fusion with modern methodologies tocreate a best fit fusion between the imaging of the patient's brainacquired before and after deformation caused by a mass occupying lesionhas occurred, for example, such as pneumocephalus. Multiple internalanatomical landmarks 140 can also be utilized to improve the best fitfunction in the overall fusion. Additional landmarks 140 may be used toimprove images further in a particular region (e.g., volume) of interestso that stereotactic targeting may be less corrupted. Some of theselandmarks 140 can also be used to verify that the fusion is accuratebefore clinical decisions are made.

The system 100 can be employed to improve the accuracy of proceduressuch as stereotactic procedures (e.g., biopsy, deep brain stimulation orother probe placement) or brain tumor surgery or any other procedurethat requires comparing an intraoperative image or postoperative imagewith a preoperative image. In the example of deep brain stimulation,compensation methods described herein can be used to compare/merge/fuseCT scans or MRIs, for example, acquired during surgery or after surgerywith preoperative imaging.

As mentioned above, one problem that can be addressed by the system 100is related to mass occupying lesions in or around the patient's brain.Examples of lesions can include pneumocephalus, tumors or the like. Suchlesions can be static although more likely the lesions can shift and/orgrow causing resulting displacement of the brain relative to the lesion.As used herein, tissue displacement generally results by the applicationor removal of force, for example, the application or removal of asurgical probe. Displacement can also include shifting which can occurwhen an initial tissue deformation shifts or moves over time. Forexample, images 120 acquired intraoperatively, the brain can bedisplaced by pneumocephalus. This imposes difficulties when comparing topreoperative imaging, which is the imaging used to select the targetsite to place a probe since the site may have shifted non-linearlyrelative to a stereotactic coordinate system. One of the purposes ofacquiring the intraoperative imaging is to assess if the probes havebeen placed in the target as intended. However, if the target isdisplaced by pneumocephalus, the intraoperative imaging may provide amisleading impression of where the probe is located (e.g., right orwrong), reducing the value of intraoperative imaging. By applying thecompensation and fusion methods described herein to images 120 by thefusion engine 110, deformation model 130, and markers 140,intraoperative imaging can be employed to better understand if theprobes are well placed or not and be utilized with greater accuracy tostereotactically guide implantation with a brain atlas or patient image.

In another aspect, the fusion engine 110 can process at least one imageevent that includes at least one of a location of a fiducial marker, aforeign body inserted in the brain, or an anatomical structure afteranatomical displacement. The image event can be correlated orco-registered with the corresponding location in the images acquiredbefore the anatomical displacement occurred or after the anatomicaldisplacement has resolved partially or completely, for example. Thefusion engine 110 can compensate for a structural shift that includes abrain shift associated with surgical procedures. In another example, thefusion engine 110 determines an initial volume for at least one selectedanatomic region in a first three-dimensional brain image for a patientand determines another volume for the selected anatomic region in asecond three-dimensional brain image for the patient.

The fusion engine 110 can compute shrinkage or growth factors from thefirst three-dimensional brain image based on a difference between theinitial volume and the another volume for the at least one selectedanatomic region and generates a mapping between the first and secondthree-dimensional brain images based on the computed shrinkage or growthfactors and at least one volume parameter for the brain to providefeedback that compensates for distortion between the first and secondthree-dimensional brain images. This can also include utilizing one ormore fiducial markers or landmarks to characterize inhomogeneities inanatomy or inhomogenities in a vector of displacement to align imagesfrom a digitized atlas and a brain image, for example.

Current technologies generally can only match well/fuse well images fromthe same patient, such as CT scan to MRI that are substantiallyequivalent. These images do not fuse well when one of the images has amass occupying lesion. Current algorithms and work flow presumes thatthe anatomy is the same between the two. The system 100 can solve thislimitation however, and allow for intraoperative imaging andpostoperative imaging to be combined and compared to preoperativeimaging. As noted above, the system 100 can perform a workflow in whichcoarse fusion of the overall brain can be initially accomplished byfusing the skull volume, extracranial structure, intracranial volume andstructures and ventricular volume as well as the subarachnoid volume,for example. Then, additional distortions can be accounted for withinternal fiducial markers and well-known landmarks 140, such as thoseresiding in a region of interest. The landmarks 140 can be used tocompensate for the inhomogeneities of structural displacement, which mayneed to be accounted for. In another example, the fusion engine 110 anddeformation model 130 can be utilized in combination with surgicalplanning software and/or hardware platforms so that the fused images(intraoperative-postoperative-preoperative) can be used to informsurgery decision-making or post-surgical evaluation. For example, alocation and/or trajectory in three-dimensional space can be computedfrom the fused image to accounts for anatomical deformation.

In another example, the system 100 can be employed to fuse patient brainimages a brain atlas and vice-versa. In some examples, the resultingfused image 134 can be used for stereotactic targeting of anatomicalsites, such as cortical or subcortical structures that cannot be wellvisualized in the patient's brain imaging. There are current clinicalplatforms that attempt at performing such procedures with inefficientand inaccurate fusion protocols. These methods are based on atlasoutlines and may be discredited by stereotactic and functionalneurosurgeons because of the high-risk that the overlaid atlas does notmatch well with the actual patient's anatomy. The system 100 solvesthese problems by utilizing image fusion with non-homogeneousmethodologies to create a best fit fusion between the patient's brainimaging and the atlas based on multiple fiducial points as well as atechnique for volume fusion of the ventricular as well as subarachnoidspaces. Multiple internal landmarks can also be utilized to improve thebest fit in the overall fusion between the image(s) and the atlas.Additional landmarks may be used to improve the fusion further in theregion of interest so that stereotactic targeting based on the fusedatlas brain can be more accurate. Some of the landmarks can also be usedby the condition to verify that the fusion is accurate before clinicaldecisions are made and/or a stereotactic procedure is performed.

Another problem with current technologies is that the current atlasesare designed for adjusting to the patient's brain in two dimensions andit assumes that the atrophic changes and differences between thepatient's brain and the atlas are homogeneous when they are often not,as illustrated below with respect to FIG. 2. These differences arevariable and inhomogeneous and in order to account for such, multiplelayers of fusion are utilized. The system 100 provides a method in whichcoarse fusion of the overall brain can be initially accomplished byfusing images of the ventricular volume as well as the subarachnoidvolume with atlas images during a rigid alignment described below withrespect to FIG. 5. A non-rigid alignment can subsequently be performedbetween the rigid alignment and the preoperative image. Then, additionaldistortions can be accounted for with internal fiducial markers andwell-known landmarks. Depending on the target of interest, theatlas-based fusion can be further improved with additional landmarks inor adjacent that same target area.

FIG. 2 illustrates an example of brain deformation in simplifieddiagrams representing brain images having different mass occupyinglesions as well as the determination of displacement vectors andfiducials. At 210 of FIG. 2, a pre-operative image of a brain is shown.At 220, after a procedure has occurred, a direction arrow shows adisplacement vector in the brain image. Such displacement could occurfor example as a bubble of air is released from an access opening in thecranium, a pneumocephalus, for example. The displacement vector canextend through an area of maximum deformation, for example. At 230, afurther non-homogeneous nature of the displacement depicted at 220 isshown. For example, a pneumocephalus within the cranium may have movedduring a procedure causing the brain to shift (e.g., deform) in anon-homogeneous manner. A displacement vector having a direction andmagnitude can be calculated based on a current level of deformation dueto the distortion and depicted along the path, shown as a displacementvector. The displacement vector can be calculated in three dimensions byutilizing a non-linear displacement function along the vector. As shown,after the displacement vector, which can be a non-linear function, hasbeen determined for a given mass occupying lesion and indicating one ormore directions and magnitude for the displacement, fiducial points canbe located (e.g., manually and/or automatically) along the displacementvector to further reduce distortions in the image. For example, thefiducial points can correspond to voxels in three dimensions.

In some examples, a volume of the bubble (pneumocephalus) can becalculated and a long axis thereof be computed. The displacement vectorcan be computed as being perpendicular to the long axis of the bubble. Agradient of the displacement along the displacement vector can also becalculated. The gradient and the displacement vector can be used tocalculate a corresponding displacement for unknown points in the regionof interest. For example, the fiducial marker can be employed to stretchor reduce boundary points in the brain image along the displacementvector and compensate for the displacement of the brain along thevector. In addition to the fiducials, landmarks within the brain canfurther be employed to reduce or increase boundary areas in order tofurther quantify the anatomical distortion. The method of FIG. 3, asdisclosed herein, will further highlight the utilization of displacementfunctions, vectors, fiducials, and landmarks to mitigate displacementdistortion in post-operative or intraoperative brain images.

In view of the foregoing structural and functional features describedabove, example methods will be better appreciated with reference toFIGS. 3, 4, and 7. While, for purposes of simplicity of explanation, themethods are shown and described as executing serially, it is to beunderstood and appreciated that the methods are not limited by theillustrated order, as parts of the methods could occur in differentorders and/or concurrently from that shown and described herein. Suchmethods can be executed by various components configured as executableinstructions stored in a computer memory (e.g., one or morenon-transitory computer readable media) and executed by a processor, forexample.

FIG. 3 illustrates an example method 300 for performing image fusionbetween brain images taken before, during, and after operativeprocedures. At 310, the method 300 includes determining a non-homogenousdisplacement function for a brain image (e.g., via deformation model 130of FIG. 1), such as after deformation (e.g., following performance of asurgical procedure). At 320, the method 300 includes determining adisplacement vector from the non-homogenous displacement function toindicate a direction for a displacement in the brain image after thedeformation has been performed (e.g., via deformation model 130 of FIG.1). At 330, the method 300 includes performing a coarse fusion in thebrain image utilizing the displacement vector (e.g., fusion engine 110of FIG. 1). At 340, the method 300 includes placing fiducial pointsalong the displacement vector to further compensate for the displacementin the brain image (e.g., via fiducial markers and landmarks 140 of FIG.1). The method 300 can also include utilizing landmarks in the brainimage in accordance with the fiducial points to further compensate forthe displacement in the brain image. This can include fusingpreoperative, intraoperative, and postoperative brain images. The method300 can also include fusing preoperative, intraoperative, andpostoperative brain images with an atlas, for example.

FIG. 4 illustrates an example method 400 for performing image fusionbetween an atlas image and brain images taken before, during, and afteroperative procedures. At 410, the method 400 includes measuring surfacevolume of a brain image (e.g., via fusion engine 110 of FIG. 1). At 420,the method 400 includes utilizing a digitized atlas in accordance withthe measured surface volume to determine an initial volume for the brainimage (e.g., via fusion engine 110 of FIG. 1). At 430, the method 400includes determining a difference in volume utilizing the initial volumein the brain image and a post-operative volume in the brain image aftera brain surgical procedure by computing shrinkage or growth factors fromthe brain image (e.g., via deformation model 130 of FIG. 1). At 440, themethod 400 includes recalibrating the digitized atlas based on a volumeparameter for the brain (e.g., via deformation model 130 of FIG. 1). At450, the method 400 includes fusing the recalibrated atlas to the brainimage utilizing the shrinkage or growth factor (e.g., via fusion engine110 of FIG. 1). The method 400 can also include utilizing one or morelandmarks to align the digitized atlas and the brain image followingalignment of one or more volume of interest. This can include increasingor decreasing target areas in the brain image via fiducial markers thatare placed on to the brain image.

FIG. 5 illustrates an example fusion engine 500 that performs rigidalignment 510 and non-rigid alignment 520 between different images ofcommon anatomical structure. In the example of FIG. 5, one of the imagesis demonstrated as a preoperative image 530 (also referred to as preopimage) and another image is demonstrated as a postoperative image 540(also referred to as post op image and/or intraoperative image). Thefusion engine 500 is configured to generate displacement data 550 withrespect to one or more points (e.g., up to all pixels or voxels) withinthe images 530 and 540.

The fusion engine 500 receives the preoperative image 530 and the postoperative image 540 and initially performs the rigid alignment 510between the two images. The rigid alignment 510 can also be considered acoarse fusion as previously described. The rigid alignment 510 isutilized to provide a general orientation between the preoperative image530 and the postoperative image 540 where a subset of image points(e.g., 16 points located at the exterior of the respective images) maybe used to generally align the two images with respect to each other(e.g., align images in the same general direction and in the samegeneral image space). In one example, the postop image 540 may have adeformation due to a surgical procedure such as an insertion of a probeor removal of tissue (e.g., tumor) as previously described. After therigid alignment 510, the fusion engine 500 can perform the non-rigidalignment 520 where non-linear vectors and transformations can begenerated to facilitate an alignment between the preop image 530 and thepostop image 540. The non-rigid alignment 520 can also be considered afine fusion as previously described.

After the non-rigid alignment 520, a mapping tool 560 determines adisplacement vector which represents a mapping between where a selectedpoint was in the preop image 530 to its displaced location in the postopimage 540. The mapping tool 560 can include a vector calculator 570 tocompute the displacement vector. As shown, the displacement data 550generated by the mapping tool 560 can be provided to a graphical userinterface (GUI) 580 to provide feedback to the user regarding theselected point. The GUI 580 can enable the user to select a plurality ofdiffering points of interest on the preoperative image 530,post-operative image 540, and/or fused image where the vector calculator570 of the mapping tool 560 computes updated displacement data 550 basedon the selected point received from the GUI 580.

The selected points can be identified manually (e.g., via fiducialpoints and landmarks previously described) and/or automatically bymapping between dynamically determined anatomical features. This couldinclude comparing features to a preop and postop grid for example anddetermining a plurality of displacement vectors for a plurality ofpoints with respect to the grids. The mapping tool 560 and vectorcalculator 570 can compute the displacement data 550 to provide adisplacement vector for each pixel or voxel in the fused image thatspecifies a magnitude and direction by which each such pixel or voxelhas shifted between the images 530 and 540. This magnitude and directionfurther can be provided as a corresponding vector in three dimensionalspace such as a stereotactic coordinate system for the patient. Afterthe respective mapping by the mapping tool 560, the displacement data550 can provide feedback via the GUI 580 to indicate how much a givenanatomical point or feature has shifted after a surgical procedureand/or how a given deformation has shifted over time. Such feedback thuscan be employed to accurately locate a given anatomical feature ordetermine a trajectory that has shifted due to the anatomicaldeformation and further be utilized for surgical intervention and/oranalysis.

Additionally, the fusion engine 500 can employ one or more predictivemodels to predict anatomical movement over time. For example, imagedatabases (not shown) can be analyzed to determine how a given type ofdeformation may shift over time. As an example, a deformation bubble(e.g., pneumocephalus) can be analyzed over time from a plurality ofimages for various patients to observe and mathematically describe how agiven bubble may shift over time and/or as further surgical procedurescommence. This can include analyzing parameters that can include probeinsertion angles, anatomical shapes, anatomical procedures employed,anatomical instruments employed, and so forth to determine how a givenbubble is initially formed and how the bubble may shift as a function oftime and related parameters. Various time/displacement shift vectors canthen be generated to update the model. The model can then be employed bythe fusion engine 500 to update the displacement data 550 over time toprovide a predicted deformation of how an anatomical structure (e.g.,for each pixel or voxel of the structure) may shift over a given timeperiod.

By way of example, the vector calculator 570 can be configured togenerate the displacement data 550 as including a volume vector fieldthat indicates on a per-voxel basis the direction and magnitude thateach respective voxel moves between each of the respective images 530and 540. For instance, the volume vector field can indicate the“aligned” (rigid) and “registered” (non-rigid) volumes. As used herein,a voxel refers to a single sample, or data point, on a regularly spacedthree dimensional grid. This data point can consist of a single piece ofdata, such as an opacity, or multiple pieces of data, such as a color inaddition to opacity, for example.

As a further example, the field vector calculator 570 can provide a 3Dfloating point field with various colors representing movement (e.g.,magnitude and direction). For example, gray areas can be used toindicate no movement (although darker or lighter grays can indicateuniform movement in all 3 axes) and colors can be used indicate movementpredominantly along one or two axes. For instance, stronger colors canindicate a greater amount of movement due to deformation. An imagecorresponding to the field vector can be provided as an output map ofthe patient anatomy to visualize the computed deformation. Differentcombinations of color scales can be utilized to depict differentfeatures of the field vector to help the user understand that extent anddirection of deformation that has occurred. Analysis of similar trendsover multiple similar image sets for one or more patients could be usedto generate the deformation model, previously described. The fusionengine 500 can interpret files in such a manner as to obtain adisplacement between the preoperative, postoperative, and intraoperativeimages on a per-pixel and/or per voxel basis.

One intermediate aspect of the registration process is to optimallyalign each intraoperative (e.g., bubble) scan 540 to each preoperative(e.g., normal) scan 530, creating a re-sampled, registered bubbledataset which, ideally, minimally differs from the normal dataset. Afterregistration, the vector field calculator 470 of the mapping tool 560can determine displacement vectors between the respective images.Various alignment tools can be employed to perform image alignmentand/or registration. One commercially available tool (e.g., the NiftyRegapplication developed at the University College London) employs commandline parameters to control behavior and execution of image alignment.Other tools may employ a configuration file to control behavior andexecution of image alignment. In general with respect to a bubbledeformation in the post op image 540, the fusion engine 500 performs therigid alignment 510 of the bubble image to normal datasets to produce a“rigid” transform representing the optimal rigid mapping of the bubbledataset to the normal dataset.

Using the “rigid” transform produced by the rigid alignment 510, thefusion engine 500 re-samples (e.g., transforms) the bubble dataset,generating a new “aligned” dataset in which the bubble data has beenrotated and/or translated to maximally overlap the normal dataset, andre-sampled onto a 3D lattice/grid similar to the normal dataset. Thefusion engine 510 then performs the non-rigid (e.g., non-linear)registration 520 of the new, aligned dataset produced at 510 to theoriginal (e.g., normal) dataset, creating a “non-rigid” transform. Usingthe “non-rigid” transform produced at 520, the fusion engine 500 createsa new “registered” dataset in which the each of the pixels or voxels arere-sampled onto a 3D lattice/grid of the normal, original datasetcorresponding to the image 530. From the registered data set, themapping tool 560 can then generate displacement vectors between selectedpoints in the post op image 540.

FIG. 6 illustrates three example images where a preoperative image isshown at 610, a rigidly aligned image at 620, and a non-rigidlyregistered image at 630. A measuring crossbar 640 represents a locationof a selected feature 642 in the preoperative image 610. At 620, afterrigid alignment between the preoperative image 610 and a postoperativeimage having a deformation bubble, the measuring crossbar 642 shows adownward vertical displacement of about 2.86 mm at 650. At 630, afternon-rigid alignment between the preoperative image 610 and apostoperative image having a deformation bubble, the measuring crossbar642 shows that the displacement of about 2.86 mm at 650 has been removedat 660. Thus, a mapping can then be performed to generate displacementdata describing how the preoperative feature shown at 642 has beendisplaced (e.g., magnitude and direction) based on the rigid alignmentand the non-rigid alignment depicted at one point at 660.

FIG. 7 illustrates an example method 700 for performing displacementmapping via rigid and non-rigid registration for different imagesacquired for a common anatomical structure of a given patient. Forexample, the images could be pre-operative and post-operative images orany images in which one of the images exhibits localized deformation ofan anatomical region, such as disclosed herein. At 710, the method 800includes accessing image data from memory that includes a first image ofan anatomical feature and a second image of the anatomical feature. Theimage modalities used to acquire the images can be the same ordifferent. The images can be two-dimensional or three-dimensionalimages. At 720, the method 700 includes performing a rigid alignmentbetween the preoperative image and postoperative image to generate arigidly aligned image (e.g., via fusion engine 110 of FIG. 1 or fusionengine 500 and rigid alignment 510 of FIG. 5). At 730, the method 700includes performing a non-rigid registration between the rigidly alignedimage and the first image (e.g., via fusion engine 500 and non-rigidalignment 520 of FIG. 5). At 740 of FIG. 7, the method 700 includesgenerating a vector field mapping based on the non-rigid alignment toprovide displacement data regarding displacement between thepreoperative image and the postoperative image (e.g., via fusion engine500 and mapping tool 560 of FIG. 5). The displacement data can include amagnitude and direction that quantifies the deformation for each commonpixel or voxel in the images accessed at 710.

Although image alignment between preoperative and postoperative imageswhich includes intraoperative images can involve first a rigid alignmentbetween images followed by a non-rigid alignment, it is possible that anon-rigid registration can be performed without first performing therigid alignment described at 830. For example, after acquiring thepreoperative image (e.g., of a patient's brain) as the first image andan intraoperative or postoperative image of the same brain in roughlythe same orientation (second image), the method 800 can perform athree-dimensional non-rigid registration of the second image to thefirst image. This non-rigid registration can be analyzed to calculate athree-dimensional vector field for which each point in the fieldindicates the direction and distance from a location in the second imageto the corresponding location in the first image.

The vector field can then be employed to provide feedback to a user(e.g., surgeon, other software, robot, medical device, and so forth)indicating compensation (e.g., distance, trajectory) required to targeta specific preoperatively identified structure, or inversely to predictthe equilibrium location of an intraoperatively/postoperativelyidentified structure. For example, the displacement data can be furthermapped into three-dimensional coordinate system of the patient (e.g., astereotactic coordinate system) that can be utilized to in a subsequentsurgical procedure to accurately access a target site in the deformedanatomical structure. As noted above, a model can be developed topredict how a structure might continue to move over time which can alsobe employed to update the feedback to the user.

As a further example, if a rigid alignment is employed before thenon-rigid registration then the method 700 can first convert the firstimage and the second image to format utilized by the fusion engine. Thiscan include pre-processing between the images, such as can includecropping the first image and second image and performing backgroundremoval to remove unnecessary data and decrease alignment time, forexample. After any pre-processing that may occur, the method 800 canthen perform rigid-registration of the second image to first image. Thiscan be achieved using a Normalized Mutual Information correlation metricwith Gradient Descent optimization and Linear interpolation, forexample. After the rigid registration, perform thenon-rigid-registration of the aligned second image to first image. Thiscan also be achieved using a Normalized Mutual Information correlationmetric with Gradient Descent optimization and Linear interpolation, forexample. Other optimization techniques could be utilized for the rigidalignment and non-rigid registration. Non-rigid registration typicallyfirst divides the images into a lattice/grid, then iteratively evaluatesand optimizes correlation metrics for each cell in the lattice usingintermediate results to adjust a continuous nonlinear mapping (vectorfield) between the second and first image, which is in turn used togenerate the correlation metric. The aligned image and vector field canthen be employed as inputs to provide guidance regarding deformation andassociated displacement between the respective images.

FIG. 8 illustrates an example system 800 for performing fusion withcompensation for brain deformation in accordance with an aspect of thepresent invention. The system 800 can include one or more generalpurpose networked computer systems, embedded computer systems, routers,switches, server devices, client devices, various intermediatedevices/nodes, and/or stand alone computer systems (desktop orportable).

The system 800 includes a processor 802 and a system memory 804. Dualmicroprocessors and other multi-processor architectures can also beutilized as the processor 802. The processor 802 and system memory 804can be coupled by any of several types of bus structures, including amemory bus or memory controller, a peripheral bus, and a local bus usingany of a variety of bus architectures. The system memory 804 can includeread only memory (ROM) 808 and random access memory (RAM) 810 as well asany other non-transitory computer readable media that can store data andinstructions executable by the processor 802. A basic input/outputsystem (BIOS) can reside in the ROM 808, generally containing the basicroutines that help to transfer information between elements within thesystem 800, such as a reset or power-up.

The system 800 can employ one or more types of non-transitory datastorage 814, including a hard disk drive, a magnetic disk drive, (e.g.,to read from or write to a removable disk), solid state memory (e.g.,flash drive) and an optical disk drive, (e.g., for reading a CD-ROM orDVD disk or to read from or write to other optical media). The datastorage can be connected to the processor 802 by a drive interface 816.The data storage 814 thus can store instructions executable by theprocessor 802 and related data for the computer system 800. A number ofprogram modules may also be stored in the data storage 814 as well as inthe system memory 804, including an operating system, one or moreapplication programs, other program modules, and program data. In theexample of FIG. 8, the data storage 814 can include a fusion system 818(e.g., the system 100 of FIG. 1 or the system 500 of FIG. 5) that isprogrammed to perform clinical imaging fusion. Such executableinstruction can be executed by the processor 802, such as when loadedfrom the RAM 810.

A user may enter commands and information into the computer system 800through one or more input devices 820, such as a keyboard, atouchscreen, physiologic input devices, biomarker readers, photo devicesand scales, card scanners, and/or a pointing device (e.g., a mouse). Itwill be appreciated that the one or more input devices 820 can includeone or more physiologic sensor assemblies transmitting data to thesystem 800 for further processing. These and other input devices areoften connected to the processor 802 through a device interface 822. Forexample, the input devices can be connected to the system bus by one ormore a parallel port, a serial port or a USB. One or more outputdevice(s) 824, such as a visual display device or printer, can also beconnected to the processor 802 via the device interface 822.

The system 800 may operate in a networked environment using logicalconnections (e.g., a local area network (LAN) or wide area network(WAN)) to one or more remote computers 830. A given remote computer 830may be a workstation, a computer system, a router, a peer device, orother common network node, and typically includes many or all of theelements described relative to the computer system 800. The system 800can communicate with the remote computers 830 via a network interface832, such as a wired or wireless network interface card or modem. In anetworked environment, application programs and program data depictedrelative to the system 800, or portions thereof, may be stored in memoryassociated with the remote computers 830.

Additionally, as disclosed herein the system 800 can be communicativelycoupled (e.g., directly or via a network connection) with a stereotacticsystem 840. The stereotactic system can employ a resulting brain atlasand/or displacement data for performing a stereotactic procedure. Forexample, a mathematical model corresponding to the brain atlas can beprovided and converted to a three-dimensional coordinate system of thestereotactic system 840 via an appropriate transform. The resultingatlas or other displacement data thus can be utilized to facilitatemanual or automated (e.g., robotic) surgery relative to an identifiedtarget site in the brain. As another example, a user can select a targetsite in a pre-deformation image (e.g., a preoperative image) and thefusion system 818 can be configured to identify the currentthree-dimensional location in a coordinate system of the stereotacticsystem 840. The fusion with the brain atlas enables a more accurateidentification of the target site as it accounts for deformation asdisclosed herein.

What have been described above are examples. It is, of course, notpossible to describe every conceivable combination of components ormethodologies, but one of ordinary skill in the art will recognize thatmany further combinations and permutations are possible. Accordingly,the disclosure is intended to embrace all such alterations,modifications, and variations that fall within the scope of thisapplication. As used herein, the term “includes” means includes but notlimited to, the term “including” means including but not limited to. Theterm “based on” means based at least in part on. Additionally, where thedisclosure or claims recite “a,” “an,” “a first,” or “another” element,or the equivalent thereof, it should be interpreted to include one ormore than one such element, neither requiring nor excluding two or moresuch elements.

What is claimed is:
 1. A system, comprising: a model to represent avolumetric deformation of a brain, stored in a memory, corresponding tobrain tissue that has been displaced or shifted by at least one ofdisease, surgery or anatomical changes; and a fusion engine, executed bya processor, to employ the model to perform a coarse fusion to align afirst three-dimensional image of a patient's brain with respect to asecond three-dimensional image of the patient's brain after a region ofthe brain has been displaced or shifted, wherein the coarse functioncomprises: determining an initial volume of at least one selectedanatomic region in the first three-dimensional image of the patient'sbrain and another volume of the at least one selected anatomic region inthe second three-dimensional image of the patient's brain; computingshrinkage or growth factors from the first three-dimensional image ofthe patient's brain based on a difference between the initial volume andthe another volume for the at least one selected anatomic region;generating a mapping between the first and second three-dimensionalimages of the patient's brain based on the computed shrinkage or growthfactors and at least one volume parameter of the model to providefeedback that compensates for distortion between the first and secondthree-dimensional images of the patient's brain; and adjusting one ormore points on a displacement vector of the model extending through theat least one selected anatomic region to compensate for spatialdeformations that occur between the first and second image of thepatient's brain based on the mapping.
 2. The system of claim 1, furthercomprising a fiducial marker in the first three-dimensional image of thepatient's brain, the fusion engine being programmed to employ thefiducial marker with the volumetric analysis of the model and apply tothe first three-dimensional image of the patient's brain to compensatefor distortion in the first three-dimensional image of the patient'sbrain for at least one spatial region of interest selected in responseto a user input.
 3. The system of claim 2, wherein the fiducial markeris selected in the first three-dimensional image of the patient's brainin response to a user input or an automated system.
 4. The system ofclaim 1, wherein the brain image includes at least one landmark orfiducial marker, the fusion engine is programmed to apply the volumetricanalysis of the model and the at least one landmark or fiducial markerto the first three-dimensional image of the patient's brain tocompensate further for distortion in the first three-dimensional imageof the patient's brain in relation to at least one other image.
 5. Thesystem of claim 1, wherein the fusion engine is programmed to fuse orco-register the first three-dimensional image of the patient's brainwith the second three-dimensional image of the patient's brain based onthe model.
 6. The system of claim 5, wherein the model and the fusionengine are employed to fuse or co-register the second three-dimensionalimage of the patient's brain with a brain atlas to provide an adjustedbrain atlas.
 7. The system of claim 6, further comprising a vectorcalculator configured to determine a displacement vector for each pixelor voxel in the second three-dimensional image of the patient's brainand transform each pixel or voxel into a coordinate system of astereotactic system.
 8. The system claim of 7, wherein the fusion engineprocesses at least one image event that includes at least one of alocation of a fiducial marker, a foreign body inserted in the patient'sbrain, and an anatomical structure after anatomical displacement.
 9. Thesystem of claim 8, wherein the image event is correlated orco-registered with a corresponding location in images acquired beforethe anatomical displacement occurred or after the anatomicaldisplacement has resolved partially or completely.
 10. The system claimof 9, wherein the fusion engine compensates for a structural shift thatincludes a brain shift associated with surgical procedures.
 11. A methodcomprising: performing a non-rigid alignment between the firstthree-dimensional image data and the second three-dimensional image datacomprising: determining an initial volume for at least one selectedanatomic region in a first three-dimensional brain image for a patient;determining another volume for the at least one selected anatomic regionin a second three-dimensional brain image for the patient; computingshrinkage or growth factors from the first three-dimensional brain imagebased on a difference between the initial volume and the another volumefor the at least one selected anatomic region; and generating a mappingbetween the first and second three-dimensional brain images based on thecomputed shrinkage or growth factors and at least one volume parameterfor the brain to provide feedback that compensates for distortionbetween the first and second three-dimensional brain images; andgenerating a vector field mapping based on the non-rigid alignment toprovide displacement data to a user regarding displacement between thefirst image data and the second image data.
 12. The method of claim 11,wherein the at least one selected anatomic region comprises at least oneof a ventricular region, a subarachnoid region, a cortical region and asubcortical region.
 13. The method of claim 11, further comprisingutilizing one or more fiducial markers or landmarks to characterizeinhomogeneities in anatomy or inhomogenities in a vector of displacementand align images from a digitized atlas and a brain image.
 14. A method,comprising: accessing a first three-dimensional image data of ananatomical feature for a patient and second three-dimensional image dataof the anatomical feature for the patient; performing a non-rigidalignment between the first three-dimensional image data and the secondthree-dimensional image data comprising: determining an initial volumefor at least one selected anatomic region in the first three-dimensionalimage data and determining another volume for the at least one selectedanatomic region in the second three-dimensional image data; computing ashrinkage or growth factor from the first three-dimensional brain imagebased on a difference between the initial volume and the another volumefor the at least one selected anatomic region; and generating a mappingbetween the first and second three-dimensional image data based on thecomputed shrinkage or growth factors and at least one volume parameterfor the brain to provide feedback that compensates for distortionbetween the first and second three-dimensional image data; andgenerating a vector field mapping based on the non-rigid alignment toprovide displacement data to a user regarding displacement between thefirst image data and the second image data.
 15. The method of claim 14,further comprising performing a rigid alignment between the first imagedata and the second image data to generate a rigidly aligned image. 16.The method of claim 15, wherein the rigid alignment or the non-rigidalignment includes at least one of a normalized mutual informationcorrelation metric, a gradient descent optimization, and a linearinterpolation between images.
 17. The method of claim 16, wherein thenon-rigid registration divides the images into a lattice and iterativelyevaluates and optimizes correlation metrics for each cell in thelattice.
 18. The method of claim 16, wherein the non-rigid registrationemploys intermediate results to adjust a continuous nonlinear mappingbetween the first image data and the second image data which is used togenerate the correlation metrics.
 19. The method of claim 14, furthercomprising generating a model which describes how the anatomical featureshifts over time and employing the model to update the vector fieldmapping over time.